I have seen this on Windows and MacOS but the attached test case seems to only reproduce the issue on MacOS

Hit MOZ_CRASH(OOM) at build/xpcom/base/nsDebugImpl.cpp:611

0|0|XUL|NS_ABORT_OOM(unsigned long)|hg:hg.mozilla.org/mozilla-central:xpcom/base/nsDebugImpl.cpp:10ad7868f3ca27cb90db9bd1d392ff4d7852a0cd|611|0x16
0|1|XUL|nsTArrayInfallibleAllocator::ResultTypeProxy nsTArray_base<nsTArrayInfallibleAllocator, nsTArray_RelocateUsingMoveConstructor<mozilla::layers::TileClient> >::EnsureCapacity<nsTArrayInfallibleAllocator>(unsigned long, unsigned long)|hg:hg.mozilla.org/mozilla-central:xpcom/ds/nsTArray-inl.h:10ad7868f3ca27cb90db9bd1d392ff4d7852a0cd|154|0x8
0|2|XUL|mozilla::layers::TileClient* nsTArray_Impl<mozilla::layers::TileClient, nsTArrayInfallibleAllocator>::InsertElementsAtInternal<nsTArrayInfallibleAllocator>(unsigned long, unsigned long)|hg:hg.mozilla.org/mozilla-central:xpcom/ds/nsTArray.h:10ad7868f3ca27cb90db9bd1d392ff4d7852a0cd|2232|0x1f
0|3|XUL|mozilla::layers::ClientMultiTiledLayerBuffer::Update(mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, mozilla::layers::TilePaintFlags)|hg:hg.mozilla.org/mozilla-central:gfx/layers/client/MultiTiledContentClient.cpp:10ad7868f3ca27cb90db9bd1d392ff4d7852a0cd|193|0x1e
0|4|XUL|mozilla::layers::ClientMultiTiledLayerBuffer::PaintThebes(mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, void (*)(mozilla::layers::PaintedLayer*, gfxContext*, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, mozilla::layers::DrawRegionClip, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, void*), void*, mozilla::layers::TilePaintFlags)|hg:hg.mozilla.org/mozilla-central:gfx/layers/client/MultiTiledContentClient.cpp:10ad7868f3ca27cb90db9bd1d392ff4d7852a0cd|118|0x16
0|5|XUL|mozilla::layers::ClientTiledPaintedLayer::RenderHighPrecision(mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, void (*)(mozilla::layers::PaintedLayer*, gfxContext*, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, mozilla::layers::DrawRegionClip, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, void*), void*)|hg:hg.mozilla.org/mozilla-central:gfx/layers/client/ClientTiledPaintedLayer.cpp:10ad7868f3ca27cb90db9bd1d392ff4d7852a0cd|356|0x29
0|6|XUL|mozilla::layers::ClientTiledPaintedLayer::RenderLayer()|hg:hg.mozilla.org/mozilla-central:gfx/layers/client/ClientTiledPaintedLayer.cpp:10ad7868f3ca27cb90db9bd1d392ff4d7852a0cd|578|0x19
0|7|XUL|mozilla::layers::ClientContainerLayer::RenderLayer()|hg:hg.mozilla.org/mozilla-central:gfx/layers/client/ClientContainerLayer.h:10ad7868f3ca27cb90db9bd1d392ff4d7852a0cd|53|0x9
0|8|XUL|mozilla::layers::ClientContainerLayer::RenderLayer()|hg:hg.mozilla.org/mozilla-central:gfx/layers/client/ClientContainerLayer.h:10ad7868f3ca27cb90db9bd1d392ff4d7852a0cd|53|0x9
0|9|XUL|mozilla::layers::ClientContainerLayer::RenderLayer()|hg:hg.mozilla.org/mozilla-central:gfx/layers/client/ClientContainerLayer.h:10ad7868f3ca27cb90db9bd1d392ff4d7852a0cd|53|0x9
0|10|XUL|mozilla::layers::ClientContainerLayer::RenderLayer()|hg:hg.mozilla.org/mozilla-central:gfx/layers/client/ClientContainerLayer.h:10ad7868f3ca27cb90db9bd1d392ff4d7852a0cd|53|0x9
0|11|XUL|mozilla::layers::ClientContainerLayer::RenderLayer()|hg:hg.mozilla.org/mozilla-central:gfx/layers/client/ClientContainerLayer.h:10ad7868f3ca27cb90db9bd1d392ff4d7852a0cd|53|0x9
0|12|XUL|mozilla::layers::ClientLayerManager::EndTransactionInternal(void (*)(mozilla::layers::PaintedLayer*, gfxContext*, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, mozilla::layers::DrawRegionClip, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, void*), void*, mozilla::layers::LayerManager::EndTransactionFlags)|hg:hg.mozilla.org/mozilla-central:gfx/layers/client/ClientLayerManager.cpp:10ad7868f3ca27cb90db9bd1d392ff4d7852a0cd|342|0xa
0|13|XUL|mozilla::layers::ClientLayerManager::EndTransaction(void (*)(mozilla::layers::PaintedLayer*, gfxContext*, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, mozilla::layers::DrawRegionClip, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, void*), void*, mozilla::layers::LayerManager::EndTransactionFlags)|hg:hg.mozilla.org/mozilla-central:gfx/layers/client/ClientLayerManager.cpp:10ad7868f3ca27cb90db9bd1d392ff4d7852a0cd|405|0xe
0|14|XUL|nsDisplayList::PaintRoot(nsDisplayListBuilder*, gfxContext*, unsigned int)|hg:hg.mozilla.org/mozilla-central:layout/painting/nsDisplayList.cpp:10ad7868f3ca27cb90db9bd1d392ff4d7852a0cd|2484|0x22
0|15|XUL|nsLayoutUtils::PaintFrame(gfxContext*, nsIFrame*, nsRegion const&, unsigned int, nsDisplayListBuilderMode, nsLayoutUtils::PaintFrameFlags)|hg:hg.mozilla.org/mozilla-central:layout/base/nsLayoutUtils.cpp:10ad7868f3ca27cb90db9bd1d392ff4d7852a0cd|4143|0xe
0|16|XUL|mozilla::PresShell::Paint(nsView*, nsRegion const&, mozilla::PaintFlags)|hg:hg.mozilla.org/mozilla-central:layout/base/PresShell.cpp:10ad7868f3ca27cb90db9bd1d392ff4d7852a0cd|6347|0x17
0|17|XUL|nsViewManager::ProcessPendingUpdatesPaint(nsIWidget*)|hg:hg.mozilla.org/mozilla-central:view/nsViewManager.cpp:10ad7868f3ca27cb90db9bd1d392ff4d7852a0cd|460|0x13
0|18|XUL|nsViewManager::ProcessPendingUpdatesForView(nsView*, bool)|hg:hg.mozilla.org/mozilla-central:view/nsViewManager.cpp:10ad7868f3ca27cb90db9bd1d392ff4d7852a0cd|395|0xb
0|19|XUL|nsViewManager::ProcessPendingUpdates()|hg:hg.mozilla.org/mozilla-central:view/nsViewManager.cpp:10ad7868f3ca27cb90db9bd1d392ff4d7852a0cd|1018|0x11
0|20|XUL|nsRefreshDriver::Tick(mozilla::layers::BaseTransactionId<mozilla::VsyncIdType>, mozilla::TimeStamp)|hg:hg.mozilla.org/mozilla-central:layout/base/nsRefreshDriver.cpp:10ad7868f3ca27cb90db9bd1d392ff4d7852a0cd|2195|0x8
0|21|XUL|mozilla::RefreshDriverTimer::TickRefreshDrivers(mozilla::layers::BaseTransactionId<mozilla::VsyncIdType>, mozilla::TimeStamp, nsTArray<RefPtr<nsRefreshDriver> >&)|hg:hg.mozilla.org/mozilla-central:layout/base/nsRefreshDriver.cpp:10ad7868f3ca27cb90db9bd1d392ff4d7852a0cd|373|0xb
0|22|XUL|mozilla::RefreshDriverTimer::Tick(mozilla::layers::BaseTransactionId<mozilla::VsyncIdType>, mozilla::TimeStamp)|hg:hg.mozilla.org/mozilla-central:layout/base/nsRefreshDriver.cpp:10ad7868f3ca27cb90db9bd1d392ff4d7852a0cd|367|0x12
0|23|XUL|mozilla::VsyncRefreshDriverTimer::RefreshDriverVsyncObserver::TickRefreshDriver(mozilla::layers::BaseTransactionId<mozilla::VsyncIdType>, mozilla::TimeStamp)|hg:hg.mozilla.org/mozilla-central:layout/base/nsRefreshDriver.cpp:10ad7868f3ca27cb90db9bd1d392ff4d7852a0cd|737|0xf
0|24|XUL|mozilla::VsyncRefreshDriverTimer::RefreshDriverVsyncObserver::NotifyParentProcessVsync()|hg:hg.mozilla.org/mozilla-central:layout/base/nsRefreshDriver.cpp:10ad7868f3ca27cb90db9bd1d392ff4d7852a0cd|639|0xe
0|25|XUL|mozilla::VsyncRefreshDriverTimer::RefreshDriverVsyncObserver::ParentProcessVsyncNotifier::Run()|hg:hg.mozilla.org/mozilla-central:layout/base/nsRefreshDriver.cpp:10ad7868f3ca27cb90db9bd1d392ff4d7852a0cd|538|0xe
0|26|XUL|nsThread::ProcessNextEvent(bool, bool*)|hg:hg.mozilla.org/mozilla-central:xpcom/threads/nsThread.cpp:10ad7868f3ca27cb90db9bd1d392ff4d7852a0cd|1236|0x6
0|27|XUL|NS_ProcessNextEvent(nsIThread*, bool)|hg:hg.mozilla.org/mozilla-central:xpcom/threads/nsThreadUtils.cpp:10ad7868f3ca27cb90db9bd1d392ff4d7852a0cd|501|0xd
0|28|XUL|mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate*)|hg:hg.mozilla.org/mozilla-central:ipc/glue/MessagePump.cpp:10ad7868f3ca27cb90db9bd1d392ff4d7852a0cd|87|0x7
0|29|XUL|MessageLoop::Run()|hg:hg.mozilla.org/mozilla-central:ipc/chromium/src/base/message_loop.cc:10ad7868f3ca27cb90db9bd1d392ff4d7852a0cd|290|0x5
0|30|XUL|nsBaseAppShell::Run()|hg:hg.mozilla.org/mozilla-central:widget/nsBaseAppShell.cpp:10ad7868f3ca27cb90db9bd1d392ff4d7852a0cd|137|0xd
0|31|XUL|nsAppShell::Run()|hg:hg.mozilla.org/mozilla-central:widget/cocoa/nsAppShell.mm:10ad7868f3ca27cb90db9bd1d392ff4d7852a0cd|694|0x8
0|32|XUL|XRE_RunAppShell()|hg:hg.mozilla.org/mozilla-central:toolkit/xre/nsEmbedFunctions.cpp:10ad7868f3ca27cb90db9bd1d392ff4d7852a0cd|913|0x13
0|33|XUL|mozilla::ipc::MessagePumpForChildProcess::Run(base::MessagePump::Delegate*)|hg:hg.mozilla.org/mozilla-central:ipc/glue/MessagePump.cpp:10ad7868f3ca27cb90db9bd1d392ff4d7852a0cd|237|0x5
0|34|XUL|MessageLoop::Run()|hg:hg.mozilla.org/mozilla-central:ipc/chromium/src/base/message_loop.cc:10ad7868f3ca27cb90db9bd1d392ff4d7852a0cd|290|0x5
0|35|XUL|XRE_InitChildProcess(int, char**, XREChildData const*)|hg:hg.mozilla.org/mozilla-central:toolkit/xre/nsEmbedFunctions.cpp:10ad7868f3ca27cb90db9bd1d392ff4d7852a0cd|744|0x5
0|36|plugin-container||||0xf0b
0|37|libdyld.dylib|start|||0x1
0|38|libdyld.dylib|start|||0x1

I can reproduce the OOM crash on Linux with layers.enable-tiles=true. This is not probably related to the summay element, but CCing TYLin just in case.

On linux with tiles enabled, in ClientMultiTiledLayerBuffer::Update() we are attempting to render a newValidRegion of < (x=-320564, y=-12243924, w=666658, h=60034712); >, with a tile size of 1024x1024, resulting in a newTIleCount of 38225456. We OOM when trying to allocate the vector of TileClient objects, let alone allocating their buffers!

Unfortunately it crashes before I can have a look at the layer / display list dump. Since it's reproducible on Linux I guess the best course of action is probably to reproduce in RR and see where that gigantic region is coming from and why it's not getting clipped to a display port.

I tried running mozregression on the testcase, and it gives the commit which enabled individual transform, obviously. By changing the testcase to use transform: scale(10, 21); rather than scale 10 21;, we get this commit.

It's not immediately clear to me what that commit does or why it could have this effect. Any ideas, Emilio?

That patch only removes two rules from the quirks mode stylesheet. In this test-case, it causes the frame tree to change in a way such as it triggers the bug, but you should be able to trigger it without those rules by adding something like the following to the test-case:

<style>
dd {
  display: block;
}
dd::before {
  content: none;
}
</style>